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Abstract 

The unit determined that “Assessment 5: Effect on Student Learning” would be best measured 
by student teachers and interns utilizing an action research activity in their clinical experience. 
Twenty four action research projects were evaluated by the Director of Student Teaching. 
Interraters blind to the Director’s scores evaluated the projects. Projects possessing a difference 
in score greater then 10 percent between the Director and Interrater were evaluated by a second 
Interrater. An Alpha Cronbach reliability test revealed the alpha coefficient for the Student 
Teaching Director and the first Interrater was .40 and .53 for the Director and the second 
Interrater. The same test was applied to the first Interrater and second Interrater and the alpha 
coefficient was found to be .74. The first term of Action Research projects were generally of 
poor quality and the scoring across raters was unreliable. These data informed recommendations 
including modifications in the assignments and more training for scorers. 
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Student Teacher Candidates’ Effect on Student Learning as Measured Through Action Research 

Projects 

The education unit has eight assessments that drive its assessment system. Assessment 
five is known as student teacher effect on student learning. In the summer of 2007 the Education 
Division faculty invested the summer in reviewing and revising unit assessments. The 
assessment system was not useful and data derived were not informing the unit. 

Over the course of six months a new assessment five emerged. After considerable 
deliberation and research on best practices in measuring student teacher effect on student 
learning, the first generation of the current model was applied to the unit’s clinical practices in 
the spring term of 2008. 

Herein you will receive an overview of literature related to student teachers utilizing 
Action Research and information regarding student teacher effect on student learning. Then, 
methodology for the first term of the study is shared. Data derived from the study are then shared 
and discussed. Ultimately recommendations, reflective of revisions already made resulting from 
data derived from the first term of assessment, are offered for consideration. 

Literature Review 

How can we best measure candidate’s effects on student learning and what effects do 
student teachers have on student learning? First, there must be an adequate data collection 
system. Also, what is the assurance that student teachers will alter practices based on the data 
they glean? Will data derived alter instruction provided by the candidates? How will we know? 
Finally, how can we cause this data to occur naturally so it is not merely an add-on to the already 
overwhelmed candidate and overworked education faculty? The sheer number of candidates 
going through teacher education programs can add to the maelstrom when trying to determine 
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effect on learners. Determining the effect of student teachers on student learning can be very 
problematic (Ada and Balach, 2007). 

The best answer to how one can determine student teacher effect on student learning 
seems to be the careful implementation of Action Research in the classroom by the student 
teacher. Action Research provides candidates with tools of systematic inquiry and beginning 
investigation skills, while providing the needed measuring stick to determine effect on student 
learning (Emery, Jumper, and Bruce, 2007). Such research has not been apparent at the 
undergraduate level in teacher education programs. It has been traditionally housed in graduate 
programs, often as the capstone, but it does have a place in candidate clinical practice. 

Candidates formulate hypotheses in regards to their effect on student learning as student teachers 
and then test this hypothesis by implementing unit guided Action Research in their clinical 
practice classroom(s). 

In establishing an Action Research project for all student teachers we must be able to 
articulate to faculty and students what it is and what it is not (Ross-Fisher, 2008). Without 
proper clarity the undertaking of such an endeavor can easily go off the proverbial tracks. 

Action Research is not experimental and it is often messy and uncertain (Goodnough, 2008). 

It is the responsibility of the education unit to ensure that the research projects don’t become too 
messy or uncertain. 

Again, what candidates incidentally learn from performing Action Research can cause 
them to feel as though they have a greater understanding of the big picture of what it means to be 
a professional educator. It offers candidates a professional identity they often don’t possess 
without engaging in such research, and it cultivates professional relationships and development 
(Warren, Doom, and Green, 2008). 
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Candidates engaged in student teaching have a view of day to day classroom operations 
like few researchers can possess. Life in the teaching trenches offers candidates the opportunity 
to develop unique strategies for meeting individual student needs. However, teachers have been 
historically reluctant to engage in research (Nonis, 2008). As education faculty, it is our 
responsibility and duty to instill in our candidates the expertise and initiative to be action 
researchers. Candidates should come to feel a sense of duty to research their classroom often and 
guide them with the data received. 

Student teachers should engage in Action Research since they can experience success in 
it, which will lead to subsequent research attempts beyond their clinical experience. Student 
teachers who struggle with the daily realities of the clinical experience are the ones who 
probably need to internalize the results of Action Research most. However, they will be the ones 
who will most likely have the greatest difficulty implementing the research and deciphering the 
results that can ultimately improve their teaching (Monroe, 2007). This is only one of several 
conundrums we face in trying to adequately equip and prepare candidates to become exemplary 
educators. 

Once student teachers collect and analyze the data garnered from Action Research where 
do they go from there? After all, how long do teacher effects persist anyway? In one study 
teacher effects on student learning are defined as, “teacher specific residuals adjusted for student 
and treatment effects” (Konstantopoulos, 2007). Considering all of the variables that affect 
student learning, what is the effect size of teacher effect on student learning? Per the study 
teacher effect is cumulative and the effects are evident beyond the current candidate/student 
experience. It would appear as though longitudinal studies hosted by the unit would be 
appropriate to best capture teacher effect on student learning, if this is the case. In other words 
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candidates may well not see the total effect on student learning through their Action Research 
snapshot view of student learning. Still, our candidates should conduct Action Research and not 
speculate beyond the collected data, other than to recommend that their effect on students be 
studied across subsequent student years in school to capture total effect size. 

Teachers as researchers can at least partially address the need for interventions intended 
to improve student performance in the classroom. Self intervention can result from analyzing 
data collected by classroom teachers in their research. This value added self-assessment has the 
potential for resulting in teachers selecting professional development in areas they determine to 
be in need of improvement. Such intervention has promise for translating into continued, positive 
effects on student learning for several years, especially in students’ early grades (McCaffrey, 
Lockwood, Mariano, and Sedodji, 2005). 

Historically, there have been calls for studies on teacher effect on student learning. 

Effect on student learning includes the amount of student time on-task and this is correlated with 
effect on student learning and candidate characteristics (Fox, 1978). Candidate characteristics do 
indeed affect student learning. Empowering candidates to analyze their effect on student learning 
through Action Research can encourage reflective practice and incite personal, positive changes 
in pedagogy and practice. 

Methodology 

The study began in the spring term of the 2007-2008 academic year. Twenty-four student 
teachers were assigned the task of completing prescribed Action Research projects in their 
assigned classroom(s). The primary study investigator met with the student teachers at the 
beginning of the term to review the multiple step research process that was to be implemented. 
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The steps included drafting a 10 day unit plan of instruction, developing and 
administering a pretest reflective of the content of the unit plan early in the term, analyzing the 
results of the pretest and revising the draft unit plan as needed, teaching of the 10 day unit plan, 
and administering a unit grand assessment with the pretest embedded therein. 

One of the secondary investigators, who served as director of student teaching, oversaw 
candidate Action Research progress across the term. Some non-Action Research assignments 
were due across the semester as well and were listed in the calendar. However, everything 
pertaining to the Action Research project was due at the end of the term. Instructions for the 
Action Research project were embedded within the unit’s 10 outcomes and were very broad. 
Instructions for the research were contained in the following: 



E-NQTEBOOK GENERAL DIRECTIONS 



These directions apply to all Discussion Board Forum assignments. You should complete all 
assignments for each discussion forum/outcome in Blackboard and save as a Microsoft Word 
document. Candidates should be prepared to provide supervisors with hard copies when 
requested. All assignments should be completed directly in the forum and submitted as 
attachments only when directions indicate that text is too big or if otherwise directed. 

Use the activity name listed as the title. Be sure to follow all directions and formats provided 
during Senior Seminar, in Handbooks, and in Course Documents in Blackboard. You are 
allowed to revise submissions so be sure that assignments are included only once. Be sure to 
post assignments and responses in the appropriate place so that discussion threads will align 
appropriately for viewing. Supervisors may be using Blackboard submissions for final grading. 
Appropriate thread alignment will insure that assignments are not overlooked. Time will not 
allow for supervisors to search for assignments not properly aligned. 

You should provide comments (two different classmates each forum) for each of the assignments 
which have been starred. Some of the individual assignments include directions for comments. 
Be sure to include these. Be sure to include assignment title in comments. Comments should be 
based on quality, correctness, creativity, relevance to the assignment, and include your personal 
observations. 



Senior Seminar Activities 
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1 . One-day Lesson Plan* 

2. Classroom Management Activity* 

3. Resume Format and Sample 

4. Code of Ethics Activity 

5. Problem Solving / Creative and Critical Thinking Activity 

6. Rubric Construction Activity * 

7. Assessment #3, 4, 6 Electronic Templates 

8. Digital Video Recording 

BPC Outcome #1- understands and demonstrates knowledge of learner characteristics. 

PAAR Summary 
Standards Table 
Class Description Activity 
Learning/Teaching Styles Inventory* 

Student Artifact* 



BPC Outcome #2 - . develops lesson plans using technology and a variety of instructional 
methods 



Standards Table 

Lesson Plans - Attach as separate days 
PowerPoint Activity* 

Smartboard Activity* 

Teaching Video 

Teacher-Made Supplemental Materials (2)* 

Student Artifact 

BPC Outcome #3 - . demonstrates mastery of content. 



Standards Table 
GACE Scores 
Transcript(s) 

Student Artifact 
Video 

BPC Outcome #4 - promotes creative and critical thinking 
Standards Table 

Creative Thinking Activity and Artifact* 

Critical Thinking Activity and Artifact* 

BPC Outcome #5 - demonstrates proficiency in writing and expression 
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Standards Table 

Newsletter - (Microsoft Word, Word Art, Clip Art/Picture)* 
Design Thank You Note* 

Candidate Selected Artifact 



BPC Outcome #6 - utilizes formal and informal methods and assessments 

Assessment #5 Effect on Student Learning - Participatory Action Research* 

Candidate Developed Rubric (may be for any assessment/activity)* 

Clinical Faculty Final Evaluations (Planning, Performance, Dispositions) 

School Faculty Evaluations (Planning, Performance, Dispositions) 

Candidate Dispositions Self Assessment 
Teacher Education Program Evaluation 
Student Artifact and completed rubric from above 

BPC Outcome #7 - demonstrates an understanding of student diversity 
Standards Table 

Differentiated Group Activity - Artifact* 

Special Needs Accommodation - Artifact* 

Learning Styles Inventory - Artifact 
Example of tables from #6* 

BPC Outcome #8 - promotes motivation and positive social interaction 
Standards Table 

Grade Book Program - Artifact* 

Games/ Awards/Certificate - Artifact* 

Classroom Management Activity (see course documents) 

Classroom Management Plan 

BPC Outcome #9 - evaluates performance and continues to grow professionally 

Standards Table 
Reflective Paper 

Professional Development Artifact* 

Sample Lesson Plan Reflection 
Resume 

Student Teaching Log (see handbook in course documents) 

BPC Outcome #10 - possesses a Christian disposition toward school, local, state, nation, and 
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world while reinforcing 

Standards Table 
Dispositions Self Assessment 

Community Involvement Summary Table (see course documents)* 

Georgia Code of Ethics Activity 

Student Discussion Board 

You may share ideas, thoughts, questions, etc. with one another in this forum. 

At the end of the term the director of student teaching printed the twenty-four Action 
Research projects from the electronic learning management system used in student teaching. 

The director and other education faculty members, serving as interraters, had devised a 10 part 
evaluation instrument. Each of the 10 parts of the evaluation tool had a possible value of 3 
points. The total possible points that could be awarded to an Action Research project were 30. 

The director of student teaching scored all twenty-four candidate projects. Then, 
interraters were randomly assigned to score the projects, blind to the director of student teaching 
scores. In cases where the scores across the director and interraters possessed a difference of 10 
percent or more a second interrater, blind to both the director’s score and the first interrater’s 
score, evaluated and scored the project. Second interraters were randomly assigned to score 
projects just as initial interraters were randomly assigned. 

Results 

Twenty-four student teachers participated in the assessment five Action Research project 
in its inaugural term, spring 2007-2008. Twenty-four action research projects were rated by the 
director of student teaching. Then, the projects were randomly assigned to other education 
division faculty interraters. Ten projects were interrated for a second time randomly by five of 
the faculty interraters who had not previously evaluated the particular projects. These ten 
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projects were interrated for a second time because the difference between the director of student 
teacher’s score and first interrater score was greater than ten percent of possible project points. 
All five second round interraters participated in the first round of interrating, but again the 
projects scored in round two were not scored by them in the first round. The total and mean of all 
the participants of the study by the rater were 317 and 13.31 respectively. The total and mean of 
all participants after the first round interrating were 315 and 13.13. The total and mean after the 
second round interrating for all participants were 276 and 1 1.50. As indicated by the data from 
the second round of interrating, eight of ten action research projects continued to possess a score 
difference greater than ten percent. This difference can be explained in several ways but it is 
most likely the result of more training being needed in scoring the projects. Between the second 
scores and the first, four of the ten score differences were larger than in the first round. The total 
difference between the director of student teaching and the first interraters was -2, while the total 
difference between the director and the second interraters was -41. The total difference between 
round one interrater scores and the second round of interrater scores was -39. An alpha cronbach 
reliability test revealed the alpha coefficient for the student teaching director and the first 
interrater was .40 and .53 for the director and the second interrater. The same test was applied to 
the first interrater and second interrater and the alpha coefficient was found to be .74. 



Table 1 



Descriptive statistics of the scores by the rater, first interater, and the second interater 



Scorer 


Number 


Mean 


sd 


Total 


Student Teh 
Director 


1 


13.21 


4.37 


317 


First 

Interrater 


6 


13.13 


4.05 


315 


Second 

Interrater 


5 


11.5 


4.63 


276 



note. N = 24 
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Table 2 



The difference means, difference totals, and alpha coefficients between the rater and 
the interater 



Differences 


Mean 


Total 


Alpha Coefficient 


Between the rater and the 
first interater 


-0.08 


-2 


0.40 


Between the rater and the 
second interater 


-1.71 


-41 


0.53 


Between the first and 
second interater 


-1.63 


-39 


0.74 



note. N = 24 



Analysis of Data 

Results were somewhat confounding on at least two levels. Overall quality of the Action 
Research projects was poor with the mean below fifty percent of points possible. Perhaps this 
could be attributed to it being the initial attempt of implementing such a project in student 
teaching. Or, it could be that the expectations were not as clear as they needed to be. Or, the fact 
that the entire project did not have to be submitted until the end of the term may have had 
something to do with the low performance. Whatever the case or reason the projects for the most 
part did not meet unit expectations. 

A second area of concern was the lack of reliability across scores given by the director of 
student teaching and the interraters. The alpha cronbach reliability needed was not evident across 
director scores and those of the interraters. It can be speculated that the director was privy to 
subjective information regarding particular candidates and factored that knowledge into the 
scoring of the projects, while the interraters had no knowledge of information beyond the 
projects themselves. In other words the director’s scores may have included data that the 



interraters could not see when they scored the projects. To further defend this hypothesis, the 
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reliability scores across interraters was very close to being at an acceptable rate to ensure 
reliability, while the discrepancy between director scores and interrater scores was remarkable. 

Conversely the mean scores across director and first interraters were amazingly similar. 
On a thirty point scale the mean difference was less than one-tenth of one point. Still, the 
standard deviations were four or higher. This dissonance between the mean scores and standard 
deviations caused the investigators to greatly appreciate having more than one way of seeing the 
data. Had the investigators only evaluated differences between mean scores they would not have 
comprehended the considerable differences within test scores between the director and interrater 
scores. 

In summary, the Action Research projects generated by the candidates in the first term of 
the study were not satisfactory. Secondly, there was no reliability across director and interrater 
scores thereby negating the assurance needed that the projects were scored in satisfactory and 
meaningful ways. Therefore, the data derived were not sufficient to inform the unit other than to 
cause it to revise the student teacher Action Research projects in substantial ways. 

Much needs to be done to improve the clarity of expectations for the projects and the reliable 
scoring of them. 

Recommendations 

In the current term (Fall, 2008) much is being done to attempt to secure better projects 
and more reliable scoring. A report outline is now evident in the electronic learning management 
system that houses the instructions for the Action Research projects. Also, a grading rubric has 
been developed and shared with the candidates. It is very specific as to what must be included 
and it articulates the desired organization method of the project. The primary investigator 
checked for candidate understanding in more meaningful ways when presenting the Action 
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Research project to them at the beginning of the term. The director who oversees the projects 
now possesses a clearer understanding of the concept of Action Research and is able to provide 
greater clarity regarding project expectations to the candidates in the second attempt. 

Additionally, each step of the Action Research project is being turned in along the way 
and these are being evaluated by the student teaching director. If the director finds any portion of 
the project submitted to be unsatisfactory the candidate must revise and resubmit that particular 
portion before proceeding. Now candidates see the following instructions online: 



Action Research - Effect on Student Learning 
Candidate Report 



Personal Information: 



Candidate’s Name 


Campus 


Semester 


Program 











Student Teaching Information: 



School 


System 


Clinical Supervisor 


School Faculty Supervisor 











Directions: 

Complete your action research report using this template. Complete the information above 
and include this page in your report. You will be submitting the elements of your report on 
different dates across the semester. Be sure to check the due dates on your course 
calendar. Submit hard copies of individual sections when they are due and place copies in 
Blackboard. You will submit a completed hard copy of the entire report at the end of the 
semester (check date in course calendar). Number the pages of your report. 

Grading: 

Individual Sections : The individual elements will be graded as they are due (see rubric in 
course documents). You may revise and update the report sections until the due date. 

Once the report section has been graded that grade will be final. Any report sections 
receiving a grade of D or F (Unacceptable) must be revised and resubmitted. Individual 
sections will not be re-graded but revisions will be considered in determining your unit’s 
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total effect on student learning. Your individual report sections will be graded by the 
Clinical Supervisor, School Faculty Supervisor, and Director of Student Teaching. 

Final Report : Your total Action Research grade will include the previous grades on all 
individual sections and a summative grade reflective of your unit’s total effect on student 
learning (see rubric). This grade will be determined through the tests, procedures, data, 
and data analysis presented in your report as well as the mechanics of grammar, spelling, 
punctuation, and format (APA). Semester grades will not be posted until an acceptable 
report has been submitted. 



Action Research 
Your Name 



I. Class Description 

II. Draft Standards-Based 10-Day Unit Plan 

-Title Page 

-Learning Objectives aligned with standards 
-Lesson format according to BPC teaching model 
-Appendix (handouts, quizzes, PowerPoints, assessments) 

III. Pretest 

-Students assigned random I.D. #s 
-Questions relevant to unit 
-Appropriate number of questions 
-Appropriate format 

-Appropriate readability level (reading level ) 

-Test administrator procedures/conditions 

IV. Pretest Results and Analysis 

-Appropriate analysis methods 
-Results clear and concise 
-Charts, graphs, tables 

V. Summary of Changes and Revisions to Unit 

-Additions, deletions, revisions of lessons based on pretest results 

-Special Needs 

-Differentiated Instruction 

-Additions, deletions, revisions of assessments 

VI. Revised Standards-Based Unit Plan 

VII. Posttest 

-Pretest questions embedded in posttest 




Student Teacher 16 



-Appropriate number of questions 
-Appropriate format 
-Appropriate readability level 
-Test administrator procedures/conditions 

VIII. Posttest Data Analysis and Interpretation 

-Aggregated class-wide results (gains/losses) included 
-Disaggregated results student by student (by student ID #s) 
-Charts, graphs, tables included 
-Summary Analysis and Interpretation 

IX. Reflection 

X. Appendix 



Education 474-475 Student Teaching 
Blackboard Discussion Board 



Forum 



Search 



Display 

Order 




T STUDENT TEACHING PERFORMANCE ASSESSMENTS 



You should complete all assessments for each discussion forum in 
Blackboard and save on a jump/finger drive or disk as a Microsoft Word 
document. Students will provide hard copies to supervisors. All 
assessments should be completed directly in the forum and submitted as 
attachments only when directed. 

Use the activity name listed as the title. You are allowed to revise 
submissions up until the due date. Be sure that assessments are posted 
only once. Be sure to post assessments and responses in the appropriate 
place so that discussion threads align appropriately for viewing. 
Supervisors will be using Blackboard submissions for final grading. 
Appropriate thread alignment will insure that assessments and 
comments are not overlooked. Time will not allow supervisors to search 
for assignments not properly aligned and/or titled. 

You should provide comments when directed. Comments should be based 
on quality, correctness, relevance to the assignment. Include your 
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personal observations. 




PERSONAL INFORMATION AND DOCUMENTATION FORMS 



Personal Information 
PAAR Information Form: 
Proof of Ins urance: 
Background Check: 
Autobiography: 

GACE Score Reports; 
Transcripts: 



AUGUST EXPERIENCE VERIFICATION AND DAILY LOG 



August Experience Verification Form 
August Experience Daily Log 



4 




PERFORMANCE ASSESSMENT ALIGNMENT (BPC 
Outcomes/Standards) 




To be completed across semester. Form provided in Course Documents. 

ACTION RESEARCH - SECTION 1 
DUE OCTOBER 2 

Draft of 10-Day Unit Plan 
Class Description 
Pretest 

Pretest Results and Analysis 



6 




7 




ACTION RESEARCH - SECTION 1.1 
DUE OCTOBER 17 

Summary of Changes and Revisions to Unit 
Revised Standards-Based Unit Plan 

ACTION RESEARCH - SECTION III 
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DUE NOVEMBER 28 
Posttest 

Posttest Data Analysis and Interpretation 




r* 



CANDIDATE 10-DAY ACTIVITIES / ARTIFACTS 

DUE NOVEMBER 6 

PowerPoint 

Smart Board Activity 

Three-Column Newsletter 

Electronic Grade Book Sample 

Professional Development 

Resume 

Classroom Management Plan 
Code of Ethics Activity 

CANDIDATE ACTIVITIES / STUDENT ARTIFACTS 



DUE NOVEMBER 13 
Teacher-Made Supplemental Materials 
Creative/Critical Thinking Activity 
Learning Styles Inventory 
Candidate Developed Rubric 
Awards Certificate 



10 




EVALUATIONS 



DUE NOVEMBER 28 
( Use directions for adding multiple images ) 
Clinical Faculty Evaluations 
-3 - Performance 
-3 - Daily Planning 
-1 - 10-Day Lesson Plan Assessment 
-2 - Dispositions 
-1 - Action Research Rubric 
School Faculty Evaluations 
-3 - Performance 
-3 - Daily Planning 
-1 - 10-Day Lesson Plan Assessment 
-2 - Dispositions 

Candidate Dispositions Self Assessment 
-2 - Beginning and End 
Teacher Education Program Evaluation 
-1 - (End of Semester) 
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11 



T STUDENT TEACHING DAILY LOG 



DUE NOVEMBER 28 



12 „ STUDENT TEACHING VIDEO 

DUE NOVEMBER 28 



13 






T ACTION RESEARCH REPORT - EFFECT ON STUDENT 



LEARNING 



14 



DUE DECEMBER 5 

EVERYTHING COMPLETED IN BLACKBOARD 
HARD COPIES OF E-NOTEBOOK DUE DECEMBER 8 



T COMMENTS 



Senior Seminar Comments and Activities 
Action Research Section I 
Action Research Section II 
Action Research Section III 
Classroom Management Plan 
Code of Ethics Comments 



t ] APPENDIX 




TEACHER EDUCATION PORTFOLIO - REQUIRED DOCUMENTS 



Required Documents - Teacher Education Portfolio 

1. Personal Information 

2. PAAR ( Demographics ) Information Form 

3. Proof of Insurance 

4. Background Check 

5. GACE Score Reports 

6. Transcripts 

7. August Experience Verification Form 

8. August Experience Daily Log 

9. PowerPoint 

10. Smart Board Activity 

11. Professional Development 

12. Learning Styles Inventory 

13. Candidate Developed Rubric 
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14. Evaluations: 

Clinical Faculty Evaluations 
3 - Performance 
3 - Daily Planning 

1 - 10-Day Lesson Plan Assessment 

2 - Dispositions 

1 - Action Research Rubric 
School Faculty Evaluations 
3 - Performance 

3 - Daily Planning 

1 - 10-Day Lesson Plan Assessment 

2 - Dispositions 

Candidate Dispositions Self Assessment 
2 - Beginning and End 
Teacher Education Program Evaluation 
1 - (End of Semester) 

15. Action Research Report 



Additional scoring training for the director and the interraters is also planned, to attempt 
to reach acceptable levels of reliability. All scorers must come to know that they only score what 
is visible to all of those who are doing the scoring. This should reduce the discrepancies evident 
in the first term of the project scoring. 

The investigators are hopeful that the added clarity provided to the candidates and 
additional training of scorers of the Action Research projects will cause the projects to become 
valuable artifacts that can provide data leading to decisions that will improve the unit. 
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